Neural Memory Networks

CONTEXT is vital in formulating intelligent classifications and responses, especially under uncertainty. In a standard feed-forward neural network (FFNN), context comes in the form of information encoded in the input vector and trained in weight parameters. However, useful information can also be present in the temporal nature of the input vectors, or from past internal states of a network. Future outputs can achieve better accuracy by observing transient trends in the input data, or by utilizing key memories from distant inputs which could be crucial to formulating a correct output. By providing a neural network with an architecture for storing and maintaining memories this additional context can be effectively leveraged. A simple implementation of memory in a neural network would be to write inputs to external memory and use this to concatenate additional inputs into a neural network. For noisy analog inputs, memory inputs pulled from Gaussian distributions can act to preprocess and filter the data. Fig. 1 shows a schematic and memory weight distribution of a FFNN with external memory augmentation.